1. Field of the Invention
The system of the present invention is related to the field of benchmarking of computer systems and computer software. Specifically the system of the present invention relates to the field of benchmarking of window based systems.
2. Related Applications
This application is related to copending U.S. patent application Ser. No. 07/498,206, filed Mar. 23, 1990, entitled "Synchronized Journaling System", and U.S. patent application Ser. No. 07/411,036, filed Sept. 21, 1989, entitled, "Method and Apparatus for Extracting Process Performance Information from a Network" and are herein incorporated by reference.
3. Art Background
The term "benchmarking" is used to describe a series of tests performed on a computer system which simulate various operations that may be performed by a user of the system. During the execution of tests, various measurements are taken. These tests are performed on the system a number of times and the results of the measurement taken are compared with optimum or other results recorded during benchmarking of similar systems.
Through the use of benchmarking, a systematic method is provided to test and measure the performance of systems in a controlled manner. With the advent of window based systems such as those manufactured by Sun Microsystems, Inc., Mountain View, Calif., it has become necessary to provide a means of testing the performance capability of such window systems and the computer software and hardware that implement the window systems. An example of early benchmarking of window systems is described in Gaylin, "How Are Windows Used? Some Notes on Creating An Empirically Based Windowing Benchmark Task", CHI Proceedings (April 1986), pages 96 to 100.
Window systems present additional factors which must be considered in evaluating system performance. For example, a distributed window system consists of multiple processes, including the application or client processes, the window server and optionally a window manager. The application processes are really client processes which are operated through the window system. The window server controls the display and mediates access to display by the client's process. It also provides a base functionality in terms of windows and cursors and determines how to draw into these windows and how to distribute events to them. Examples of window systems include the X11.TM. (X11 is a trademark of the Massachusetts Institute of Technology) which supports the X11 protocol (See Sheifler, Gettys, "The X Windows System", ACM Transactions On Graphics, Vol. 5, No. 2 (April 1986) pp. 79-109) and X11/NeWS system, available from Sun Microsystems, Inc., Mountain View, Calif., which supports both the X11 and NeWS.TM. window server protocols (NeWS is a trademark of Sun Microsystems, Inc.) (See Shaufler, "X11/NeWS Design Overview, Proceedings of the Summer 1988 User Conference, (1988) pp. 23-35).
The client process is typically linked with a "toolkit" which implements the user interface functionality and provides the means for the client process to tailor the user interface to the client process. The toolkit determines how the window systems menus look, how users interact with control areas and how control areas are organized. Furthermore, the toolkit provides a programmatic interface that allows developers of the application programs to implement sophisticated windowing applications. An example of a toolkit is the XView.TM. toolkit (XView is a trademark of Sun Microsystems, Inc.) available from Sun Microsystems, Inc., for the X11NeWs Window System. For further information, see Jacobs, "The XView Toolkit, An Architectural Overview", 3rd Annual Technical Conference, (1989).
The window manager may be a separate process as is typically the case with the X11 protocol or it may be part of a window server as is the case with respect to the X11/NeWS window system where the window manager is a collection of processes that reside in the window server. The window manager controls the workspace, the window that occupies the entire screen and determines the workspace menu window and icon placement.
In a kernel-based window system, the kernel is modified to include an event manager and a means to contain window state information. This portion of the system is in charge of the input/output devices and of distributing events to application processes. The kernel-based window system also consists of a user interface toolkit which provides the applications' menus, control areas and drawing capabilities. An example of a kernel-based window system is the SunView.TM. window system (SunView is a trademark of Sun Microsystems, Inc., see Sun Microsystems Inc., SunView 1 Programmer's Guide, revision A, (5/9/88) and Sun Microsystems, Inc., SunView 1 Systems Programmer's Guide, revision A, (5/9/88)).
In order to provide an accurate benchmark of the system, it is desirable to simulate as closely as possible a user session in which the user performs a variety of tasks on the system. A technique was developed to provide this type of benchmark referred to as journaling. Journaling is a method for recording events reflective of a series of user actions in a session file referred to as a "journal file". The file is subsequently read and the actions initiated and executed in the sequence recorded in the file, thereby providing an automated means for simulating the user session (See, e.g., CAPBAK/UNIX Terminal Session Capture and Playback for UNIX Systems, Software Research Incorporated, (1989)).
Window based systems introduce additional actions to consider when developing a benchmark for the system. A window based system consists of a number of input devices used concurrently and number of processes which interact with one another in a pre-determined sequence in order to produce the desired effect, such as selecting a client or application process from a menu or moving the cursor from one window to another. In an attempt to maintain the proper sequence of execution, a time delay between various actions is recorded in the journal file in addition to the other information recorded to simulate a user session. By scaling the time between events, it is possible to propagate events through the system faster on playback than on record. However, playing events faster than the speed at which they were initially recorded can lead to unexpected behavior. This unexpected behavior arises because of race conditions caused by one process executing prior to the uncompleted execution of another process, the final state of which is utilized by the subsequent process.
A method was developed using special events marking the locations in the journal file where previously initiated processing must be completed before initiating a subsequent process. These special events are referred to as synchronization events and are typically defined at locations where multiple processes are exchanging state. These synchronization events are put into the journal sessions during the recording phase. On playback, the journaling mechanism waits for each of these synchronization events to occur before proceeding through the journal file and initiating subsequent events in the window system. (See Islam, Ingoglia, "Testing Window Systems", Interfaces, Systems and People Working Together, 28th Annual Technical Symposium, ACM, Washington, D.C. (1989) pp. 95-103; and copending U.S. patent application Ser. No. 07/498,206, filed Mar. 23, 1990, entitled "Synchronized Journaling System").
There are a variety of parameters or metrics employed to measure the performance of a computer system, such as the speed of execution of a process and the output generated during execution of a process. Another metric used to measure the performance of a computer system is the measurement of the size of the working set of a process. The working set is the number of memory pages referenced during execution of a process. The working set size is indicative of the processing speed because the more pages referenced, the greater occurrence of paging. Paging is a time consuming process in which a page of information is swapped into and/or out of fast memory.
The working set is measured at predetermined time intervals and the working set size is compared to measurements taken at prior intervals to determine the performance of the system. One such system is described in copending U.S. patent application Ser. No. 07/411,036, filed Sept. 21, 1989, entitled "Method and Apparatus for Extracting Process Performance Information From a Network". Time-dependent measurements (i.e., measurements performed during predetermined time intervals) perform adequately in traditional non-window-based systems because the execution sequence is primarily controlled by the system providing a predictable temporal system in which the time based measurements can be set to measure the system performance during different portions of process execution.
However, in window-based systems, the system is user interactive and event driven. Thus, the system no longer executes in a manner that is time predictable and time-based measurements do not always occur in predictable portions of process execution. In addition, some window systems, such as X11/NeWS, include shared libraries which are jointly referenced by multiple processes. In window systems which employ shared libraries, it is also preferred that the performance on the system relevant to usage of shared libraries is measured.